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Abstract 

Most discovery systems for silent failures work in two phases: a continuous monitoring phase that 
detects presence of failures and a localization phase that identifies the faulty element(s). This separa- 
tion is important because localization requires more significant resources than detection and should be 
initiated only when a fault is present. 

Detection, which we focus on, uses a schedule of probe packets. We propose ways to improve the 
system efficiency by addressing both the selection of appropriate objectives for detection time, based on 
costs associated with particular failures, and the efficient design of schedules, which satisfy objectives 
with minimum probing overhead. Our work unifies the treatment of SUM and MAX objectives and of 
stochastic and deterministic schedules and allows us to compare and relate these alternatives. 

We define memoryless schedules - a subclass of stochastic schedules which is simple and suitable 
for distributed deployment. We show that optimal memorlyess schedules can be efficiently computed 
by convex programs (for SUM objectives) or linear programs (for MAX objectives), and surprisingly 
perhaps, are guaranteed to have expected detection times that are not too far off the (NP hard) stochastic 
optima. 

Deterministic schedules allow us to bound the maximum (rather than expected) cost of undetected 
faults, but like stochastic schedules, are NP hard to optimize. We propose novel efficient determinis- 
tic schedulers with analytic performance guarantees and simulate them on real networks. By relating 
performance to the computed memoryless optima, we can see that on many instances, our deterministic 
schedules are nearly optimal. 
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1 Introduction 



Prompt detection of failures of network elements is a critical component of maintaining a reliable network. 
Silent failures, which are not announced by the failed elements, are particularly challenging and can only be 
discovered by active monitoring. 

Failure identification systems iPlOl [TT1 [T3l [T2l typically work in two phases: First detecting presence 
of a fault and then localizing it. The rational behind this design is that detection is an easier problem than 
localization and requires light weight mechanisms that have little impact on network performance. Once the 
presence of a fault is confirmed, more extensive tools which may consume more resources are deployed for 
localizing the fault. Moreover, in some cases, it is possible to bypass the problem, by rerouting through a 
different path, quicker than the time it takes to pinpoint or correct the troubled component. 

A lightweight failure detection mechanism, which relies on the existing infrastructure, uses probe pack- 
ets or probes that are sent from certain hosts or between OD pairs along the existing routing infrastruc- 
ture (see Figure [TJ. The elements we monitor can be physical links ifTOll . combination of components 
and paths ifTTTl . or logical components of network elements like the forwarding rules in the switches of a 
software-defined network lfl2l . If one of the elements on the probe path fails, the probing packet will not 
reach the destination, and in this case the probe has detected a failure. 

The goal is to design schedules which optimize the tradeoff between failure detection time or more 
generally, the cost or expected cost associated with failures, and the probing overhead. We work with 
continuous testing where failures may occur at any time during the (ongoing) monitoring process, and we 
would like to detect the failure soon after it occurs. This is in contrast to one-time testing, where tests starts at 
some point in time and the goal is to detect the presence of an existing failure. Continuous monitoring comes 
in many flavors: deployment can be centralized or distributed across the network and may require following a 
fixed sequence of probes {deterministic schedules) or allow for randomization {stochastic schedules). There 
are also several natural objectives: MAX e objectives set a different detection time target for each element and 
aim to meet all these targets with minimum overhead whereas SUM e objectives aim to minimize a weighted 
sum (average) of detection times. 

We unify the treatment of these diverse methods and objectives in a common framework which we 
then use to develop scheduling algorithms and to study performance - whilst stronger objectives are often 
desirable, it is important to quantify their increased cost. 

We define a simple and appealing class of stochastic schedules which we call memoryless schedules. 
Memoryless schedules perform continuous testing by invoking tests selected independently at random ac- 
cording to some fixed distribution. The stateless nature of memoryless scheduling translates to minimum 
deployment overhead and also makes them very suitable in distributed settings, where each type of test is ini- 
tiated by a different controller. We show that the optimization problem of computing the probing frequencies 
under which a memoryless schedule optimizes a SUM e objective can be formulated as a convex program 
and when optimizing MAX e objectives, as a linear program. In both cases, with respect to either SUM e 
or MAX e objectives, the optimal memoryless schedule can be computed efficiently. This is in contrast to 
general stochastic schedules, over which we show that the optima are NP-hard to compute. Surprisingly 
perhaps, we also show that the natural and efficiently optimizable memoryless schedules have expected de- 
tection times that are guaranteed to be within a factor of two from the respective optimal stochastic schedule 
of the same objective. Moreover, detection times are geometrically distributed, and therefore variance in 
detection time is well-understood, which is not necessarily so for general stochastic schedules. 

Our convex program formulation for the optimal probing frequencies for SUM e objectives generalizes 
Kleinrock's classic "square-root law" to resource allocation problems with subset tests. Kleinrock's law, 
which applies to the special case of singleton tests (where each test can detect the failure of a single el- 
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emenjj), states that the optimal probing frequencies are proportional to the square root of the weighting 
frequencies 0. 

Applications requiring hard guarantees on detection times or failure costs or implementations requiring 
a fixed schedule prompt us to consider deterministic scheduling. Deterministic schedulers, however, are less 
suitable for distributed deployment and also come with an additional cost: the optimum of an objective on 
a deterministic schedule can exceed the expectation of the same objective over stochastic schedules. We 
study the inherent gap (which we call the D2M gap) between these optima. We show that for deterministic 
scheduling, performance within SUM e or MAX e objectives further depends on the dependence on time 
(average or maximum). While all variants are NP hard, there is significant variation between attainable 
approximation ratios for the different objectives: The stricter MAX e and SUM e objectives, requiring good 
performance at any point in time, can not be approximated better than logarithmic factors whereas the 
weakest, which consider average performance over time, have factor 2 approximations. 

Building on this, we efficiently construct deterministic schedules with approximation and D2M ratios 
that meet the analytic bounds. Our random tree (R-Tree) schedulers derive a deterministic schedule from the 
probing frequencies of a memoryless schedule, effectively "derandomizing" the schedule while attempting 
to loose as little as possible on the objective in the process. We show that when seeded, respectively, with a 
SUM e or MAX e optimal memoryless schedule, we obtain deterministic schedules with approximation ratio 
of O(log^) for the strongest SUM e objective and ratio 0(log^ + logn) for the strongest MAX e objective, 
where n is the number of elements and £ is the maximum number of tests that can detect the failure of a 
particular element. We also present the Kuhn-Tucker (KT) scheduler which is guided by the Kuhn Tucker 
conditions on the convex program computing the optimal SUM e memoryless schedules and is geared to 
SUM e objectives. The KT scheduler adapts gracefullt to changing priorities, for example in response to 
change traffic patterns in the network, and allows for a seamless transition. 

Finally, we evaluate the different schedulers on realistic networks of two different scales: We use both 
a globe-spanning backbone network and a folded-Clos network, which models a common data center archi- 
tecture. In both cases, the elements we are testing are the network links. For the backbone, our tests are 
the set of MPLS paths and for the Clos network we use all routing paths. We demonstrate how our suite 
of schedulers offers both strong analytic guarantees, good performance, and provides a unified view on at- 
tainable performance with respect to different objectives. We also demonstrate how our theoretical analysis 
explains observed performance and supports educated further tuning of schedulers. 

An important contribution of our work is the unified general treatment of multiple settings and objectives 
which were studied singularly in previous work, and offering a precise understanding of their relations and 
tradeoffs. Our work facilitates an informed choice of the proper objective for the problem at hand and 
efficient algorithms to compute or approximate the optimal solution. 

The paper is structured as follows. In Section [2] we present our model, general stochastic and deter- 
ministic schedules, and explain the different objectives. Memoryless schedules are introduced in Section [3] 
Deterministic scheduling is discussed Section |4j followed by the R-Tree scheduler in Section [5] and Kuhn- 
Tucker schedulers in Section [6] Experimental results are presented in Section [8} extension of the model to 
probabilistic tests in discussed in Section|7} and related work is discussed in Section[9] 

2 Model 

An instance of a test scheduling problem is specified by a set V of elements (which can be thought of 
as network elements or links) of size n with a weight function p (which can be thought of as priority or 
importance of the elements) and a set S of tests (probe paths) of size m. For i G [m] , test i is specified by 

'This also works for sets of non-overlapping elements. 



3 



Figure 1 : Network and elements covered by ab and cd origin-destination tests. 



a subset S{ C V of elements. A failure of element e can be detected by probing i if and only if e G Si, that 
is, if and only if test i contains the failed element]^] We use t e to indicate the number of tests which include 
element e and £ = max e £ e . 

Continuous testing is specified by schedules which generate an infinite sequence a of tests. The schedule 
can be deterministic (where it is just a finite cycle of tests) or stochastic, in which case, it is defined as a 
distribution over a countable set of deterministic sequences. Note that this general definition covers also 
adaptive stochastic schedules in which the probability distribution of the tests at time t depends on the 
actual tests preformed prior to time t. We also introduce memoryless schedules, which are a special subclass 
of stochastic schedules, in which the probability distribution of the tests is fixed over time. 

2.1 Objectives 

Objectives for a testing schedule aim to minimize a certain function of the number of tests invoked until a 
fault is detected. (We essentially measure time passed until the fault is detected by the "number of probes" 
required to discover it If the probing rate is fixed this is indeed the time.) Several different natural objectives 
had been considered in the literature. Here we consider all these objectives through a unified treatment which 
allows us to understand how they relate to each other and how they can be computed or approximated. 

The detection time T (T (e, t) for element e at time t by a schedule er is the expected time to detect a fault 
to element e that occurs at time t. If the schedule is deterministic, then T <T (e, t) = min^> e G s ah+t . If the 
schedule is stochastic, we take the expectation over sequences 

Ttr(e,t) = E CT [mine £ s ah ] . 

We classify natural objectives as MAX e , when aiming to minimize the (weighted) maximum over ele- 
ments, or as SUM e when aiming to minimize a weighted sum over elements. Both types of objectives are 
defined with respect to the weight function p over elements. 

The weighting, or priorities of different elements, can capture the relative criticality of the element 
which in turn, can be set according to the volume or quality of service level of the traffic they handle. With 
the SUM e objectives, the weights can also correspond to estimated probability that elements fail, in which 
case the weighted objective capture the expected detection time after a failure, or to the product of failure 
probability of the element and cost of failure of this element, in which case the weighted objective is the 
expected cost of a failure. With the MAX e objectives we can use p e = l/r e , where r e is the minimum 
desired detection time for a failure of element e, or the cost of a unit of downtime of element e. We then aim 
to minimize the maximum cost of a failing element. In the sequel we assume that weights are scaled so that 

With SUM e , YsePe = ancl witn MAX e, max e p e = 1. 

2 This can be extended to the case where failures are detected with some positive probability. 
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We further distinguish between objectives based on their dependence on time: average or maximum. To 
facilitate treatment of the different objectives we define the operators M e and E e , which perform weighted 
maximum or average over elements, and M t and E t , which perform maximum or average over time. More 
precisely, for a function g of time or a function / over elements: 

M t \g] = sup g(t) 
t>i 

n— >oo H 

M e [/] = maxp e /(e) 

e 

E e [/] = J>/(e) 

e 

An application of the operator E t requires that the limit exists and M t requires that g(t) is bounded. 

When the operators are applied to the function T(e, t), we use the shorthand M t [e|cr] = M t [T^e, t)], 
Et[e\<r] = E t [T cr (e,t)], M e [t\<r] = M e [T er (e, t)], E e [t\a] = E e [T cr (e,t)]. For a particular element e, 
Mt[e|<r] is the maximum over time t of the expected (over sequences) number of probes needed to detect 
a failure of e that occurred in time t, and E t [e|<r] is the expectation (over sequences) of the (limit of) the 
average over time t of the number of probes needed to detect a failure of e that occurred in time t. For a 
particular time t, M e [t\<r] is the (weighted) maximum over the elements of the expected detection time of 
a failure at t, and E e [i|<x] is the weighted sum over the elements of their expected detection times at t. We 
consider all objectives that we can obtain from combinations of these operators. The operator pairs M e and 
M t (maximum over time or over elments) and E e and E t (average of expectation) commute, but other pairs 
do not, and we obtain six natural objectives, three MAX e and three SUM e . 

MAX e objectives: In order of decreasing strictness, the three MAX e objectives are M e [M t [e|<r]], the 
weighted maximum over elements of the maximum over time of the detection time. In this objective 
and the following ones M e [E t [e\<r]], the weighted maximum over elements of the average over time, and 
Et[M e [t|<r]], the average over time of the maximum element at that time. We shorten notation as follows. 

M e M t [cr] = M e [M t [e|o-]] = supp e T (T (e, t) 

e,t 

M e E t [cr] = M e [Et[e|<x]] = maxp e E t [e|cr] 

e 

1 k 

E t M e [a] = E t [M e [t|o-]] = lim - V maxp e T (T (e, t) . (1) 

h^oo h * — ' e 

SUM e objectives: In order of decreasing strictness, the three SUM e objectives are E e [Mt[e|tr]], the 
weighted sum over elements e of the maximum over time t of detection time T(e,t), M t [E e [i|(r]], the 
maximum over time of the weighted sum over e, and E e [Et[e|<r]], the weighted sum over elements of the 
average over time. We similarly shorten notation as follows. 

E e Mt[cT] = E e [M t [e\a]} = ^PeMt[e\a] 

e 

M t E e [a] = M t [E e [t\a}} = sup Vp e T,(e,t) = supE e [t|cr] 
E e E t [a] = E e [E t [e|<r]] = ^p e E t [e|o-] 
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When the schedule a is clear from context, we omit the reference to it in the notation. There clearly 
exist ininnite sequences of probes, deterministic or stochastic, over which our objectives are not defined. 
The M e M t , E e M t , and M t E e are defined when M t [e] is defined for all elements and the M e E t and E e E t 
when Et[e] is defined for all elements. The EfM e requires that the limit in Equation ([!]) exists. Formally, 
we define a schedule to be valid if for all elements e, M t [e] and E t [e] are well defined, and for all tests i 

the relative frequency of probing i converges, that is, the limit lim^oc, fe^' - ^ exists. We limit our 
attention only to valid schedules. 

2.2 Relating and optimizing objectives 

The following lemma specifies the basic relation between the objectives. Its proof is straightforward. 
Lemma 2.1. For any schedule a, 

SUM e : E e M t [a] > M t E e [a] > E e E t [a] (2) 
MAX e : M e M t [cr] > E t M e [a] > M e E t [a] (3) 

For any objective we want to find schedules that minimize it. We denote the infimum of the objective 
over determinitic schedules by the prefix opt/), over memory less schedules by optj\/, and over stocastic 
schedules by opt. For example for the objective M e E t , optc>-M e E t is the infimum M e E t over deterministic 
schedules. Since memoryless and determinitic schedules are a subset of stochastic schedules, the determin- 
istic or the memoryless optima are always at least the stochastic optimum: For any objective opt^, > opt 
and opt M > opt. 

According to Q and ([3]), the deterministic, memoryless, or stochastic optima of each objective satisfy 
the same relations. We show that for stochastic (and subsequently also for memoryless) schedules, all three 
objectives within each category (SUM e or MAX e ) are equivalent. We therefore use the streamlined notation 
opt-SUM e and opt-MAX e for the stochastic optima of all three SUM e or MAX e objectives and similarly 
optjv/-SUM e and optjy/-MAX e for the memoryless optima. 

Lemma 2.2. . 

opt-E e E t = opt-M t E e = opt-E e M t = opt-SUM e (4) 
opt-M e E t = opt-E t M e = opt-M e M t = opt-MAX e (5) 

Proof. We provide a high level proof sketch (full proof is in Appendix [A]): We take a stochastic schedule 
with certain E e E t and "randomize" the start time by replacing it with a distribution of shifts of the schedule 
with "uniform" start time. The new stochastic schedule has for each element e, the same T(e, t) for all times 
t which equals to E t [e\ of the original schedule. The technical difficulty handled in the full proof is that the 
schedule is infinite and we therefore need to work with a prefix. □ 

We show that optimizing any of our SUM e or MAX e objectives, over either stochastic or deterministic 
schedules is NP hard (proof in Appendix [A]) 

Lemma 2.3. Computing any one of the following optima is NP hard: optr)-E e Et, optrj-MtE e , optrj-E e Mt, 
opt-SUM e , optn-M e M t , optn-M e E t , opt D -E t M e , and opt-MAX e . 
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2.3 Deterministic versus stochastic 

The distinction between objectives within each of the MAX e and SUM e groups only matters with determin- 
istic scheduling. For an instance and objective, we attempt to understand the relation between the deter- 
ministic and stochastic optima. For deterministic MAX e objectives, the compatison is to opt-MAX e and for 
SUM e objectives, it is to opt-SUM e . 

We show that on all instances, the deterministic E e E t is equal to opt-SUM e . Deterministic E e M t and 
M e M t , however, are always strictly larger (proof in Appendix [A]) . 

Lemma 2.4. 

opt D -E e E t = opt-SUM e (6) 

opt D -E e M t > 2opt-SUM e - 1 (7) 

opt D -M e M t > 2opt-MAX e - 1 (8) 

Additional relations, upper bounding the determinstic objective by the stochastic objective follow from 
relations with memoryless optima which are presented in the sequel. 

3 Memoryless schedules 

Memoryless schedules are particulary simple stochastic schedules specified by a probability distribution q 
on the tests. At each time, independently of history, we draw a test i G [m] at random according to q 
(i G [m] is selected with probability qi) and probe i. It is easy to see that in memoryless schedules detection 
times are distributed geometrically. We show that memoryless schedules perform nearly as well, in terms of 
expected detection time, as general stochastic schedules. For notational convenience, we use the distribution 
q to denote also the memoryless schedule itself. 

We first show that all SUM e objectives and all MAX e objectives are equivalent on any memoryless 
schedule. 

Lemma 3.1. For any memoryless schedule q, 

E e M t [q] = M t E e [q) = E e E t [q) = Y.7T = SUM e [q] 
M e M t [q] = E t M e [q] = M e E t [q] = max ^ = MAX e [q] , 

e Qe 

where Q e = £ i|ees . H- 

Proof. The detection time of a fault on e via a memoryless schedule is a geometric random variable with 
paramter Q e . In particular, for each element e, the distribution T(e, t) are identical for all t and its expec- 
tation, 1/Q e , is equal to M t [e] and E t [e\. From linearity of expectation, the E e E t , M t E e , and E e M t are all 
equal to YlePe/Qe- Similarly, M e M t , M e E t , and E t M e are all equal to max e § L . □ 

3.1 Memoryless Optima 

We show that the memoryless optima with respect to both the SUM e and MAX e objectives can be efficiently 
computed. This is in contrast to deterministic and stochastic optima, which are NP hard. 

Theorem 3.1. The optimal memoryless schedule for SUM e objectives, that is, the distribution q such that 
SUM e [q] = optm-SUMe is the solution of the convex program ^ (Figure^. 
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Figure 4: Computing SUM e and MAX e optimal memoryless schedules. 

The optimal memoryless schedules with respect to the MAX e objectives can be computed using an LP. 

Theorem 3.2. The optimal memoryless schedule for MAX e , that is, the distribution q which satisfies MAX e [q] 
optM-MAX e is the solution of the LP ([10]) (Figure^. 

Singletons instances: When each test is for a single element, the optimal solution of the convex program 
Q has the frequencies of each element proportional to the square root of p e 0, that is, q e = ■ s /p e / ^ e y/pl. 
The SUM e optimum for an instance with weighting p is 



opt M -suM e (p) = Y / - = Y,vp~eY,^ 



<ii) 



In contrast, the solution of the LP ( |T0| ) has optimal probing frequencies q e proportional to p e , that is, q e 



Pe/YlePe and the MAX e optimum is optj\/-MAX e (p) = max e ^ = J2 e Pe- 
3.2 Memoryless versus Stochastic 

For both SUM e and MAX e objectives, the optimum on memoryless schedules is within a factor of 2 of the 
optimum over general stochastic schedules. 

Theorem 3.3. 



opt-SUM e < opt M SUM e < 2opt-SUM e 
opt-MAX e < opt m -MAX e < 2opt-MAX e 



(12) 
(13) 



Proof. The left hand side inequalities follow from memoryless schedules being a special case of stochastic 
schedules. To establish the right hand side inequalities, consider a stochastic schedule and let ft be (the limit 
of) the relative frequency of test i. We have 



Mt[e] > E t [e] > ^ Pe 

e L l~,i\eesi w 

Therefore, the average over elements ^ e ^ must be at least half the optimum of d9b and the maxi- 



mum max P 



2E, 



must be at least half the optimum of ( 10 1 



□ 
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The following example shows that Theorem 3.3 is tight in that the "2" factors are realizable. That 



is, there are instances where the memoryless optimum is close to a factor of 2 larger than the respective 
stochastic optimum. 

Lemma 3.2. For any e, there is an instance on which 

opt M -MAX e = opt M -SUM e > (2 - e)opt-MAX e = opt-SUM e 

Proof. The instance has n elements, corresponding n singleton tests, and uniform prioritiesp e . The optimal 
memoryless schedule, the solution of both ( fT0| ) and ([9]), has q e = 1/n and M t [e] = E t [e] = n for each 
element. The optimal deterministic schedule repeats a permutation on the n elements and has M t [e] = 
M e [t] = n and E t [e] = E e [t] = (n + l)/2 for all e,t. The optimal stochastic selects a permutation 
uniformly at random every n steps and follows it. It has M t [e] = E t [e] = (n + l)/2 for all elements. □ 

3.3 Memoryless versus deterministic 



Since a deterministic schedule is a special case of a stochastic schedule, from Theorem 3.3 the memoryless 



optimum is at most twice the deterministic optimum. The proof of Lemma 3.3 shows: 
Lemma 3.3. For any e, there is an instance on which 

opt M -MAX e = opt M -SUM e = opt D -M e M t = opt D -E e M t = opt D -E t M e > 
(2 - e)opt D -M e E t = opt D -E e E t = opt D -M t E e = opt-MAX e = opt-SUM e 

That is, for the weaker SUM e and MAX e deterministic objectives, a gap of 2 is indeed realizable, 
meaning that it is possible for the deterministic optimum to be smaller than the respective memoryless 
optimum. For the strongest objectives, E e M t for SUM e and M e M t for MAX e , we show that the deterministic 
optimum is at least the memoryless optimum: 



Lemma 3.4. 



optM-SUM e < optD-E e M t 
optM-MAX e < opt D -M e M t 



Proof. Similar to the proof of Theorem 3.3 Consider a deterministic schedule and let qi be (the limit of) 
the relative frequency of test i. We have MJel > ^ Pe . □ 

The other direction, upper bounding the deterministic optimum by the memoryless optimum, is exten- 
sively dealt with in the following sections. 

4 Deterministic scheduling 

For a deterministic schedule and an objective, the approximation ratio, which for consistency we also refer 
to as the D2D, is the ratio of the objective on the schedule to that of the (determinsitic) optimum of the same 
objective. We are ultimately interested in efficient constructions of deterministic schedules with good ap- 
proximation ratio and in quantifying the cost of determinism, that is, asking how much worse a deterministic 
objective can be over the stochastic objective. The D2S ratio (or D2S in short) of a deterministic schedule 
as the ratio of the objective on the schedule to that of the stochastic optimum of the same objective. 

Since both deterministic and stochastic optima are NP hard to compute, so is the approximation ratio 
and the D2S. We therefore relate the performance of a deterministic schedule to the memoryless optimum, 
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using the D2M, which we define as the ratio of the objective on the schedule to that of the memoryless 
optimum of the same objective. The D2M of any given schedule can be computed efficiently by computing 
the memoryless optimum. Moreover, it can be used to bound the hard to compute D2D and D2S: The 



relation D2S < D2M < 2 D2S follows from Theorem [33] T he D2D is upper bounded by D2D < 2 D2M 
(For E e M t and M e M 4 , we have D2D < D2M see Lemma |3.4[). Accordingly the D2M > 1/2 and for E e M t 
and M e M t we have D2M > 1. The optimum D2M is the minimum possible over all schedules. We refer to 
the supremum of optimum D2M over instances as the DIM gap of the scheduling problem. 

Our proposed constructions of the R-Tree and Kuhn-Tucker deterministic schedulers are presented in 
detail in the following sections. Our analytic results are summarized in Table [T] The lower bounds on the 



D2M gap are established in Lemma 4.2 through example instance on which the optimum D2M is large. 



The lower bound on approximability is established in Lemma 4.3 Both lower and upper bounds for E e E( 
and M e E t are established in Lemma 4.1 E e M t D2M upper bound in Theorem |5.1| and M e M t D2M in 
Theorem 15 .21 



objective 


scheduling D2M 


D2M gap 


approximability 


E e E t 


1 


1 




M e E t 


1 


1 




E e M 4 


O(lnm) 


O(lnm) 




M e Mt, E t M e 


0(logn + \og£) 


Q(logre), Q(m) 


£7 (log re) 



Table 1: The D2M guaranteed by our constructions (in particular, this implies upper bound on the D2M), 
and lower bounds on the D2M gap and on efficient approximability. 



4.1 OPT D -E e E t and OPT D -M e E t 

For the objectives E e E t and M e E t , which are respectively the weakest SUM e and MAX e objectives, we 
show that the deterministic optimum is at most the memoryless optimum. Moreover, we can efficiently 
construct deterministic schedules with D2M arbitrarily close to 1 (and thus approximation ratio of at most 
2). 

Lemma 4.1. 

opt D -M e E t < opt M -MAX e 
opt D -E e E t < opt M SUM e 

and for any 5 > we can efficiently construct deterministic schedules with M e Et or E e Et D2M < (1 + 5). 

Proof. For any e, for a long enough run of the memoryless schedule q, there is a positive probability that 
for all elements, the average T(e, t) is within (1 + e) of its expectation E t [e\q]. When the schedule is long 
with respect to max e E t [e|g] we can obtain a deterministic schedule which "cycles" through it while that 
property. The schedule has M e E t < (1 + e)optM-MAX e . □ 

4.2 Lower bounds on D2M gap 

In contrast, for the strongest objectives, M e M t , M e E t , and E e M t , we construct a family of instances with 
asymptotically large optimal D2M. 

Lemma 4.2. There is a family of instances with m tests and n elements such that each element participates 
in £ tests with the following lower bounds on DIM: The EfM e -D2M (and thus M e Mf-D2M) il(lnn) and 
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Q(m). The E e Mt D2M is f2(log £). Moreover, these instances can be realized on a network, where elements 
are links and tests are paths. 

4.3 Approximability lower bounds for opt D -M e M t and opt D -E t M e 

Lemma 4.3. The problems opt£>-M e Mt and optrj-EtM e are hard to approximate to anything better than 
ln(n). 

Proof. When p is uniform, opt£)-M e M t is equivalent to set cover - an approximation ratio for opt£>-M e M t 
implies the same approximation ratio for set cover ifTTTl . which is hard to approximate 151 . 

This also extends to opt£>-E t M e , again using uniform p. A minimum set cover of size k implies a 
schedule (cycling through the cover) with E t M e of k. Also, a schedule with E t M e at most k means that 
M e [t] < k for at least one t, means there is a cover of size k. □ 

5 R-Tree schedules 

We present an efficient construction of deterministic schedules from a distribution q and relate detection 
times of the deterministic schedule to (expected) detection times of the memoryless schedule defined by q. 

We can tune the schedule to either MAX e or SUM e objectives, by selecting accordingly the input fre- 
quencies q as a solution of (10 1 or Q. We then derive analytic bounds on the D2M of the schedules we 



obtain. 

The building block of random tree (R-Tree) schedules is tree schedules, which are deterministic sched- 
ules specified by a mapping of tests to nodes of a binary tree. A tree schedule is specified with respect to 
probing frequencies q and has the property that for any test, the maximum probing interval in the determin- 
istic schedule is guaranteed to be close to 1/qi- However, if we do not place the tests in the tree carefully 
then for an element covered by multiple tests the probing interval can be close to that of its most frequent 
test, but yet far from the desired (inverse of) Q e = 2~2i\ e es- ?*• Therefore, even when computed with respect 
to q which solves ([9]), the tree schedule can have E e M t and E e E t D2M ratios Q(£). 

We define a distribution over tree schedules obtained by randomizing the mapping of tests to nodes. We 
then bound the expectation of the E e M t and E e E t (when applied to q which solves (|9]>) and M e M t (when 



applied to q which solves < \10[ ) over the resulting deterministic schedules. Given a bound on the expectation 
of an objective, there is a constant probability that a tree schedule randomly drawn from the distribution 
will satisfy the same bound (up tp a small constant factor). An R-Tree schedule is obtained by constructing 
multiple tree schedules drawn from the distribution, computing the objectives on these schedules, and finally, 
returning the best performing tree schedule. Note that even though the construction is randomized, the end 
result, the R-Tree schedule, is deterministic, since it is simply a tree schedule. 

Specifically, lets take SUM e as an example, we apply the R-Tree schedule construction several times 
with q's solving ([9]). The tree with the best E e M t has 0(log(f)) E e U t D2M and the tree with the best E e E t 
has a constant E e E t D2M. Furthermore we can also find a tree which satisfies both guarantees. 

Theorem 5.1. A deterministic schedule with E e Mt D2M ratio of 0(\ogl) and a constant E e Et D2M ratio 
can be constructed efficiently. 



The theorem is tight since from Lemma 4.2 the E e M t D2M gap on some instances is Q,(\ogl), and 
therefore, we can not hope for a better dependence on ^ 

For MAX e , we show that when we apply the R-Tree schedule construction to q which is the optimum 



of ( pLOp , we obtain a deteministic schedule with 0(log£ + logn) M e M t D2M. 



3 As a side note, recall that according to |6]l there exist schedules with E e Ej D2M close to 1, so with respect to E e E t this only 
shows that we can simultaneously obtain a E e Mt D2M that is logarithmic in I and at the same time a constant E e Et D2M. 
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Theorem 5.2. A deterministic schedule with M e Mt D2M ratio of O (log £ + logra) can be constructed 
efficiently. 



From Theorems 5.1 and |5.2| we obtain the following upper bounds on the D2M gap and efficiently 



construct deterministic schedules satisfying these bounds (summarized in Table [TJ. 

optD-EeMt = 0(log£)opt-SUM e 
optD-M e M t = 0(log£ + logn)opt-M e M t 

5.1 Tree schedules 

A tree schedule is a deterministic schedule guided with frequencies q where probes to test i are spaced 
[l/qi, 2/qi) probes apart. When qi has the form % = 2 _J , test i is performed regularly with period 2 J . 

Assume for now that qi = 2~ Li for positive integer Lj for all i. We map each % to nodes of a binary 
tree where i is mapped to a node at level Lj and no test can be a child of another. This can be achieved by 
greedily mapping tests by decreasing level - we greedily map tests with level Li = 1, then tests with Li = 2 
and so on. Once a test is mapped to a node, its subtree is truncated and it becomes a leaf. 

From this mapping, we can generate a deterministic schedule as follows: The sequence is built on 
alternations between left and right child at each node. Each node "remembers" the last direction to a child. 
To select a test, we do as follows. First visit the root and select the child that was not visited previous time. 
If a leaf, we are done, otherwise, we recursively select the child that was not previously visited and continue. 
This until we get to a leaf. We then output test i. This process changed "last visit" states on all nodes in 
the path from the root to the leaf. It is easy to see that if a leaf at level L is visited once every 2 L probes. 
An example of a set of frequencies, a corresponding mapping, and the resulting schedule is provided in 
Figure [5] 

If probabilities are of general form, we can map each test according to the highest order significant bit 
(and arbitrarily fill up the tree). When doing this we get per-test ratio between the actual and desired probing 
frequencies of at most 2. Alternatively, we can look at the bit representation of qi- separately map all "1" 
positions in the first few significant bits to tree nodes. In this case the average probing frequency of each test 
is very close to qi but the maximum time between probes depends on the relation between the tree nodes 
to which the bits of test i are mapped to. The only guarantee we have on the maximum is according to the 
most significant bit 2~r iog2 ( 1//|?i )"l. Under "random" mappings the expectation of the maximum gets closer 
to the average. 

Singleton tests. For a given instance, the best D2M we can hope for is when the deterministic scheduler is 
able to perform each test in precise intervals of l/qi, which results, for singletons instances, in maximum 
probing interval of l/qi. Tree schedules achieve this when qi = 2~ Li for all i. A deterministic tree schedule 
for singletons has D2M that is at most 2, and therefore, for all our objectives, the D2M gap is at most 2. 

The M e M t D2M gap and the E e M t D2M gap, however, are exactly 2. Consider an instance with two 
elements one with priority p\ = 1 — e and the other with priority p2 = e. 



Consider M e M t . The optimal memoryless schedule ( [TO] ) has q\ = 1 — e and q^ = e and max e p e M t [e] = 
1. Whenever there are at least two elements with positive priorities, any deterministic scheduler has M t [e] > 
2 for all elements. Therefore, the M e M t of any deterministic schedule is at least 2 and the D2M is at least 2. 

Consider E e M t . The optimal memoryless schedule (|9]) has qi = ^1^^- and qi = yj=^F and the 

E e M t = pi/qi + = — e + V^) 2 ~ 1- A deterministic schedule has M t [e] > 2 for both elements 

and thus E e M t = piM t [l] + p 2 Mt[2] = 2. It folows that the E e M( D2M ration is > 2 - e for any small 
e > 0. 
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qi=Q2 = 1/4, fts = <?4 = 1/8, q 5 = q& = <?7 = 1/16, q s = <?9 = 1/32 



L 


schedule 


2 


1 2 x x 


3 


1 2 3 4 1 2 x x 


4 


1234125x12341267 


5 


1234125812341267 
1234125912341267 



Figure 5: Mapping tests to nodes of a binary tree to produce a deterministic schedule. The table shows the 
level-L schedule for L = 2,3, 4, 5. The full deterministic schedule cycles through the level-5 schedule. 



Several deterministic schedules for singletons with ratio at most 2 (and better than 2 when possible 
for the particular instance, in particular when priorities are small) were previously proposed El |4j. Tree 
schedules are of interest to us here because they can be "properly" randomized to yield good performance 
in our treatment of general instances. 

5.2 Random tree schedules 

Consider an instance and a memoryless schedule with frequencies q. We assume that qi have the form 2~ Li 
for positive integers Li (this is without loss of generality as we can only look at the highest order bit and 
loose a factor of at most 2). We construct a tree schedule for q by mapping the tests to nodes randomly as 
follows. We process tests by increasing level. In each step (level), all tests of the current level are randomly 
mapped to the available tree nodes at that level. After a test is mapped to a node, its subtree is truncated. 

For each level N (which can be at most the maximum Li), we can consider the level-N schedule, which 
is a cyclic schedule of length 2 N . The schedule specifies the probes for all tests with level Li < N, and 
leaves some spots "unspecified". 

We now specify the level-iV schedule of the tree. Consider a completion of the tree to a full binary one 
with 2^ leaves (truncate everything below level N). Associate with each leaf a a binary number a which 
contains a at digit % (from right to left, i.e. the least significant digit correspond to the child of the root and 
the most significant digit corresponds to the leaf itself) if the zth child on its path from the root is a left child. 
We refer to a as the position of leaf a. 

We construct the sequence by associating test i with all leaf descendants of the node containing it, and 
with all the positions of the sequence corresponding to these leaves. Putting it in another words the level- N 
schedule of the tree cycles through the leaves a at level- N (of the completion of the tree) according to the 
order defined by a and probes the test associated with each leaf. A test with = 2~ Li is probed in regular 
intervals of 2 L \ The first probe is distributed uniformly at random from [0, 2 Li — 1]. 
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Level- N schedules constructed from the same mapping for different depths N are consistent in the 
following sense: The level N' > N schedule is 2 N ~ N repetitions of the level- N schedule in terms of the 
tests specified by a level- N schedule (those with level Li < N) and also specifies tests with N < Li < N'. 

5.3 Analysis of R-Tree schedules 

For a single element e, we analyze the expected (over our randomized construction of a deterministic tree 
schedule) maximum probe interval in the deterministic schedule. We show 

Lemma 5.1. The expected maximum is Q(\ogl e )/Q e , where l e = {i \ e S Si} . I.e., for any element e, 
E a i go [maxt T(e, t)] < c \og(£ e ) /Q e , where T(e, t) is the elapsed time from time t until e is probed. 

Proof. Given a level ./V schedule, we say that a subinterval of [0, 2^ — 1] is hit by a test if contains a leaf of 
the test. We say it is hit by an element e if it is hit by at least one test containing the element. 

Consider a particular element e. We now look only at the tests which include the element. To simplify 
notation, let qi, i G [£ e ] be the frequencies of these tests, let Q = Qi-> an d (/max = maxj qi. 

We consider the schedule for some level N G [log 2 (l/<2Wa;)> maxj Lj\. We will make a precise choice 
of N later on. 

Note that any interval of size > 1/ <? m ax must be hit by the test with maximum frequency. We are now 
looking to bound the distribution of the size of the largest interval that is not hit. 

Consider now a subinterval C [0, 2^ - 1] of size D < l/q max . We can assume that D = 2? for some j 
and the interval left endpoint is an integral multiple of D. 

We upper bound the probability that the interval it is not hit by e. The probability that it is not "hit" by a 
test with frequency qi is qiD. These probabilities of not hitting the interval by different tests are negatively 
correlated: conditioned on some of the tests not hitting the interval, it only makes it more likely that other 
tests do hit the interval - hence, the probability that the interval is not hit by any test is at most the product 
Y\i(l — qiD), which in turn is bounded from above by n«(l — QiD) < exp(— Yl QiD) = exp(-QD). 

We now upper bound the probability that there exists at least one subinterval of size D = 2 J and left 
endpoint that is an integral multiple of D, that is not hit by any test. We do a union bound on 2 N / D intervals 
of this property and this probability is at most 

^exp(-QD). (14) 

Note that if using D = f , this upper bounds the probability that there exists an interval of size x that is 
not hit (without restrictions on endpoints). This probability, in terms of x, is 

2 N+1 

exp(-Qx/2) (15) 

x 

We now restrict our attention to a subset S of the tests which satisfy qi > M-. We have Qs = J2ies 3* — 
Q/2. We now look only at the tests in S. since this is a subset of the tests that include e, it is sufficient to 
bound the expectation of the largest open interval with respect to these tests. Since the highest level in S 
is N = |~log 2 (24/Q)] < 1 + log 2 (4/Q), we can look at the level N schedule. We substitute this N and 
Qs > Q/2 in ( fT3| ) we obtain that the probability of an empty interval of size x is 

^exp(-xQ/4). (16) 
xQ 



For x = 8m^ e /Q in ( fT6| ), we obtain a bound of l/(£ e ln£ e ) < 1/2 (for £ e > 2, £ e = 1 is already 
covered as q max )- 
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We can now obtain an upper bound on the expectation of the maximum empty interval by summing over 
positive integers i, the product of interval size (i + l)x and an upper bound on the probability of an empty 
interval of at least size ix, for positive integer i, we obtain that the expectation is O(x) = (1 /Q)0 {In i e ). 

□ 



Proof of Theorem HI 

Proof. We start with frequencies q and build a deterministic tree schdule using our randomized construction. 
We show that the expected E e M t of the deterministic schedule that we obtain is at most 0(ln£) times the 
E e Mi of the memoryless schedule for q. To obtain our claim, we take q to be the optimum of (|9]>. 



We apply Lemma 5.1 The lemma shows that for each element e we have E a i g [m&x t T(e,t)] < 



clog(£ e )/Q e . Now we take a weighted sum over elements using p. We get that, 

E e ^ Pe E a i g [maxT{e,t)\ < 2_^p e — 



This is equivalent to, 



E alg E e ^ Pe [maxT(e, t)] < ^ clog(4)^ < clog(W) Yl 7T 

This implies that with probability at least 1/2 (over the coin flips of the algorithm) we get a deterministic 
schedule whose E e M t is 2clog(^ max ) ^ e It follows that the E e M t D2M ratio is at most 2clog(£ max ). 

We now show that the E e E( D2M ratio of a random tree schedule is constant with constant probability. 
Using the same reasoning as in the proof above for E e M t it suffices to show that for each e, E a i g E t [T(e, t)] < 
c/Q e . 



Fixing e and an arbitrary time t, as in the proof of Lemma 5.1 we can easily derive that Pr[T(e, t) > 
D] < exp —Q e D. In particular we get that Pr[T(e, t) > i/Q e ] < exp(— i). So the fraction of times t in 
which T(e,t) > i/Q e is at most exp(-i). It follows that E a i g E t [T(e, t)] < (2/Q e ) ^ exp(-i) < c/Q e 
for some constant c. □ 



Proof of Theorem H31 

Proof. We use ( fT6] ) in the proof of Lemma |5.1| For an element e, the probability of an empty interval of 
size at least x is at most |g exp(— xQ/4). Using x = D e = 8(lnn + ln£ e )/Q we obtain that there is an 
interval empty of tests for e of length at least D e with probability at most 1/n 2 . 

By the probability union bound over the elements we get that the probability that for all e there is no 
empty interval of length more than D e is at least 1 — 1/n. □ 

6 Kuhn-Tucker inspired schedule 

The Kuhn-Tucker conditions on the optimal solution of our convex program (|9]> translate to 
being balanced. 
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We suggest a deterministic greedy heuristic for SUM e , illustrated in Algorithm [T] which is based on 
that. For each element e, we track x e which is the elapsed number of probes since e was last probed. We 
then choose the test i with maximum X^ e |ees Pe{x e + I) 2 - 

We conjecture that the KT schedule has E e Et which is at most twice the optimal. Viewing the quantitiy 
^2 e Pe{x e + l) 2 as "potential" the average reduction in potential is the E e E t of the sequence. We do not 
provide bounds on the approximation ratio, but test this heuristic in our experiments. 

Algorithm 1 Kuhn-Tucker (KT) schedule 

1: function BEST-TEST 

2: V <- 

3: for s G S do 

4: y <- 

5: for e G s do 

6: y ^y + x[e] 

7: if (y > v) then 

8: b <— s; v y 

return b > best test 

9: function KT-SCHEDULE(V, p, S) 
10: for e € V do 
11: sc[e] <- 

12: while True do 

13: S BEST-TEST() 

14: Output S 

15: for e G s do 

16: x[e] <- 



The KT scheduler can be deployed when priorities are modified. This is in contrast to other schedulers 
which pre-compute the schedule . 

7 Extension to probabilistic tests 

A useful extension of our model allows for a probability 7r e j that depends on i and e that a failure to e is 
found with test i. We assume that different probes invoking the same or different tests are independent. 
Probabilistic tests can model ECMP (equal cost multi-paths) and transient (inconsistent) failures: Transient 
failures are modeled by a fixed probability n e i G (0, 1] of packet loss. Tests under ECMP are modeled by Sj 
being a unit flow between the origin and destination that defines a probability distribution over tests, where 
the "flow" traversing e is 7r e j. 

With probabilistic tests, we may as well use stochastic schedules, in particular, memoryless schedules, 
which also offer strong guarantees on the variance of detection times. Our models and results for memoryless 
schedules have straightforward extensions to probabilistic tests. The convex program for optjy/-SUM e can 
be modified to incorporate probabilistic tests if we replace in (|9]> X^ees, 9* by Yli ^eiQi- The LP for optjv/- 
MAX e can be modified by replacing in ( flO| ) for each element e X^| e es ^ by Yli Weili- 
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SUM e in memoryless schedulers: 



E P E/ in deterministic schedulers: 



algorithm 


GN-U 


GN-P 


GN-Z 


Clos 


Convex 


95.66 


59.72 


25.92 


32.02 


LP 


105.29 


68.77 


118.38 


32.02 


Uniform 


229.16 


72.27 


260.46 


33.00 


SAMP SC 


111.56 


82.70 


86.17 


32.00 


SAMP KT 


108.54 


61.45 


86.17 


32.00 



algorithm 


GN-U 


GN-P 


GN-Z 


Clos 


SC 


60.27 


49.42 


51.52 


16.50 


KT 


58.04 


33.93 


14.63 


16.50 


RT CON 


66.43 


49.21 


17.92 


30.76 


RT LP 


85.47 


63.81 


88.07 


31.00 


RT-S CON 


57.87 


46.91 


18.69 




RT-S LP 


59.70 


50.24 


87.47 





M t E e in deterministic schedulers: E e M t in deterministic schedulers: 



algorithm 


GN-U 


GN-P 


GN-Z 


Clos 


algorithm 


GN-U 


GN-P 


GN-Z 


Clos 


SC 


70.43 


62.08 


93.80 


16.50 


SC 


124.93 


109.46 


114.29 


32.00 


KT 


70.04 


62.08 


20.36 


16.50 


KT 


130.11 


93.02 


35.34 


32.00 


RT CON 


72.23 


56.53 


24.65 


36.05 


RT CON 


180.00 


179.91 


53.40 


144.14 


RT LP 


95.81 


73.34 


113.47 


36.20 


RT LP 


319.12 


261.65 


269.01 


146.70 


RT-S CON 


60.08 


50.02 


23.38 




RT-S CON 


121.24 


103.91 


42.61 




RT-S LP 


63.09 


53.82 


96.67 




RT-S LP 


123.35 


107.42 


183.89 





Table 2: SUM e objectives. Table shows expected time with memoryless schedules (same for all SUM e 
objectives) and E e Et < MtE e < E e Mt on different deterministic schedulers. 



8 Experimental Evaluation 

We evaluated the performance of our schedulers for testing for silent link failures in two networks. The first 
is a backbone network (denoted GN in the sequel) of a large enterprise. We tested 500 of the network links 
with 3000 MPLS paths going through them. 

The second network we considered is a (very regular) folded Clos network (denoted Clos) of 3 levels 
and 2048 links. On this network we considered all paths between endpoints. The Clos network is a typical 
interconnection network in data centers. 

For the Clos network, we only considered uniform weights (priorities), meaning that all links are equally 
important. For the GN network, we considered uniform weights (denoted GN-U), weights that are propor- 
tional to the number of MPLS paths traversing the link (GN-P, where P designates popularity), and Zipf 
distributed weights with parameter 1.5 (GN-Z). 

On these four networks (links and paths with associated weights), Clos, GN-U, GN-P, and GN-Z, we 
simulated our schedulers and evaluated their performance with respect to the different objectives. 

Memoryless schedulers: We solved the convex program Q for SUM e objectives and the LP Q for MAX e 
objectives to obtain optimal memoryless probing frequencies q. These optimization problems were solved 
using Matlab and CVxQ 

We compared these optimal memoryless schedules to other memoryless schedules obtained using three 
naive selections of probing frequencies: the first is uniform probing of all paths (Uniform), the second 
is uniform probing of a smaller set of paths that cover all the links (SAMP SC), and the third is probing 
according to frequencies generated by the Kuhn-Tucker schedule (SAMP KT). 

4 See http://cvxr.com/cvx/. 
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Figure 6: Distribution of time to detect a fault of a link in the backbone network with equal weights. 

The performance of these schedules, in terms of the expected detection times T(e, t) is shown in Table|2] 
(SUM e objective) and Table[3](MAX e objective). The schedulers optimized for one of the objectives, SUM e 
or MAX e , clearly dominate all others with respect to the objective it optimizes. We can see that while on 
some instances the alternative schedulers are close to the optimal one, on others performance gaps are 
substantial. In particular, even with respect to the scheduler optimizing the other objective. This shows the 
importance of selecting appropriate objective and optimizing for it. 

To better understand the difference between the SUM e and MAX e objectives, we depicted in Figure|6]a 
reverse CDF of the average expected time to detect a failure in a link of the backbone network with uniform 
weights (GN-U). One can see that the LP, which optimizes the worst-off link, has a smaller maximum value 
but the average, which corresponds to the area below the curve, is larger than for the convex program, which 
optimizes the average link. 

Deterministic schedulers: Next we evaluated the performance of the deterministic schedulers. Here, 
T(e,t), the wait from time t till the next path containing e is scheduled, is deterministic. We used two 
different implementation of the R-Tree algorithm (Section [5]). In the first, the algorithm was seeded with 
the frequencies computed by the LP (RT LP) or by the convex program (RT CON) when applied to the full 
set of paths. We discuss the second implementation in the sequel. We also implemented the Kuhn-Tucker 
(KT) scheduler (Section [6]), and a simple greedy Set Cover algorithm (SC) which was previously proposed 
for the M e M( metric (minimum set cover is the optimal deterministic scheduler for M e Mi when priorities 
are uniform). This scheduler produces a cyclic sequence consisting of a set cover produced by the greedy 
approximation algorithm for set cover. 

Table [2] shows the values of all SUM e objectives for the different memory less and deterministic sched- 
ulers and Table|3]shows the same for the MAX e objectives. One can easily see that indeed all results follow 
the ratios between the different objectives (for example M t E e > E e E t or M e E t < E t M e ). Moreover, while 
the SC algorithm works well for equal weights it performs fairly bad across all objectives for GN-Z. 

When priorities are uniform (as in Clos and GN-U), minimum set cover produces the optimal determin- 
istic schedule for M e M t and E t M e . We can see that indeed SC performs very well. When priorities are 
highly skewed, however, its performance deteriorates. 

The KT scheduler performed well on the SUM e objectives, which it is designed for. Because its adaptive 
nature, rather than precomputing a fixed schedule, it is recommended for applications where priorities are 
changing on the go, such as when the priority of an element correspond to current traffic levels traversing 
the element. 
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Figure 7: Distribution of time to detect a fault: RT LP and RT CON (deterministic) vs. memoryless over 
GN-Us. 

Our R-Tree schedulers (RT CON and RT LP) did not perform well on some of the instances, and in some 
cases, performed worse than SC and KT The reason, as the analysis shows, is the logarithmic dependence 
on I, which in our case, is the number of tests used to cover the link in the solution of the LP and convex 
programs. The original collection of paths turned out to have high redundancy, where subpaths have many 
alternatives and the fractional solvers tend to equally use all applicable paths. We can see evidence for this 
fragmentation in Figure [6] 

To address this issue, we seeded the R-Tree algorithm with respective solutions of the LP and Convex 
programs applied to a modified instance with a pre-selected small subset of the original paths. The subset 
was picked so that it contains a cover of the links and also tested to ensure that the objective of the optimiza- 
tion problem does not significantly increase when implementing this restriction. On those instances, tests 
which constitute a set cover of the links and produced by the greedy approximation algorithm, performed 
well. We denote the respective schedulers obtained this way using the LP and convex solutions, by RT-S LP 
and RT-S CON. 

The results of this experiment are included in Tables [2] and [3] We can observe that this heuristic im- 
proves the performance of the R-Tree algorithm substantially for all objectives. The value of SUM e (for 
the memoryless schedule produce by solving the convex program on this subset) has increased from 95.66 
to 104.33 and the value of MAX e had increased from 132.05 to 142.01. We leave the question of how to 
choose the subset to best balance the loss in the objective of the memoryless schedule with the gain in better 
derandomization for further research. 

Memoryless vs. Deterministic: In many practical scenarios, memoryless schedulers have considerable 
advantage, due to their statelessness. However, due to their stochastic nature, we only have guarantees on 
the expectation. Deterministic schedulers, on the other hand, require a centralized implementation but can 
provide worst case guarantees regarding the time (or weighted cost) until detecting a failure. We demonstrate 
this issue by illustrating, in Figure [7] the distribution over the links of the backbone graph of the maximum 
detection time in the deterministic R-Tree scheduler, M t [e] , and the 99th percentile line for the memoryless 
schedulers (elapsed time to detection in 99% of the time). Figure [8] shows the same data for the schedulers 
RT-S LP and RT-S CON which were derived after restricting the set of paths over which optimization was 
performed. One can see that when there are strict requirements on worst-case detection times, deterministic 
schedules dominate. 
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MAX e in memoryless schedulers: 



IVLE+ in deterministic schedulers 



algorithm 


GN-U 


GN-P 


GN-Z 


Clos 


Convex 


221.53 


21.81 


6.85 


32.02 


LP 


132.05 


12.65 


2.67 


32.02 


Uniform 


2787 


12.73 


249.28 


34.00 


SAMP SC 


143.00 


53.65 


72 


32.00 


SAMP KT 


243.00 


22.74 


72 


32.00 



algorithm 


GN-U 


GN-P 


GN-Z 


Clos 


SC 


72.00 


43.41 


48.17 


16.50 


KT 


122.00 


11.97 


4.28 


16.50 


RT CON 


162.00 


20.15 


5.31 


40.61 


RT LP 


173.90 


18.92 


2.91 


40.53 


RT-S CON 


92.50 


22.15 


4.90 




RT-S LP 


71.50 


22.16 


1.90 





E t M e in deterministic schedulers M e M t in deterministic schedulers 



algorithm 


GN-U 


GN-P 


GN-Z 


Clos 


algorithm 


GN-U 


GN-P 


GN-Z 


Clos 


SC 


142.99 


54.02 


50.80 


32.00 


SC 


143.00 


95.02 


113.00 


32.00 


KT 


234.30 


34.02 


4.54 


32.00 


KT 


243.00 


35.65 


9.00 


32.00 


RT CON 


345.85 


55.26 


6.12 


147.71 


RT CON 


468.00 


85.72 


24.00 


257.00 


RT LP 


531.12 


65.31 


7.05 


156.80 


RT LP 


833.00 


97.79 


13.95 


225.00 


RT-S CON 


182.78 


42.69 


6.01 




RT-S CON 


184.00 


50.00 


14.00 




RT-S LP 


142.00 


43.22 


3.21 




RT-S LP 


142.00 


54.00 


5.00 





Table 3: MAX e objectives. Table shows expected time with memoryless schedules (same for all MAX e 
objectives) and M e Et < E<M e < M e M< on different deterministic schedulers. 



9 Related work 

This basic formulation of failure detection via probes applies in multiple network scales, from back bones 
to data centers |fT3l fTTTl . A recent application is testing of all forwarding rules in a software-defined net- 
work fl2l . Beyond the detection of network failures, the fundamental optimization problems we study 
models classic and emerging resource replication and capacity allocation problems. 

Previous considerations of the detection problem for network failures focused on MAX e objective when 
all elements have equal importance (uniform priorities) lfT3l [TT1 [T2l . In this particular case, deterministic 
scheduling is equivalent to finding a minimum size set of tests which covers all elements, which is the classic 
set covering problem. The optimal memoryless schedule is a solution of a simplified LP, which computes 
an optimal fractional cover. In practice, however, some elements are much more critical than others, and 
the uniform modeling does not capture that. Ideally, we would like to specify different detection-time tar- 
gets for failures which depend on the criticality of the element. A set cover based deterministic schedule, 
however, may perform poorly when elements have different priorities and there was no efficient algorithm 
for constructing good deterministic schedules. Moreover, the SUM e objectives, which were not previously 
considered for network failure detection application, constitute a natural global objective for overall per- 
formance, for example, when elements have associated fail probability, SUM e minimization corresponds to 
minimizing expected failure detection time. 

The special case of singletons (each test contains a single element) received considerable attention and 
models several important problems. The SUM e objective on memoryless schedules is the subject of Klein- 
rock's well know "square root law" |9|. Scheduling for Teletext [2] and broadcast disks CO, can be for- 
mulated as deterministic scheduling of singletons. Both E e E( and M e Mt objectives were considered. Our 
Kuhn-Tucker inspired scheduling for MAX e generalizes a classic algorithm for singletons [7J which has 
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Figure 8: Distribution of time to detect a fault: RT-S LP and RT-S CON (deterministic) vs. memoryless over 
GN-U. 



a factor 2 approximation for the E e E t 0. Bar-Noy et al. (3J0 established a gap < 2 between the op- 
timal deterministic and memoryless schedules, this is in contrast to the difficulty of general subset tests, 
where we show that gaps can be asymptotic. Interestingly, however, even for singletons, M e M t optimal 
deterministic scheduling is NP hard [3). several approximation algorithms were proposed for deterministic 
scheduling (HE! 13). m particular, Bar-Noy et al. J3j|4l proposed tree-schedules, which are an ingredient 
in our R-Tree schedule constructions, as a representation of deterministic schedules. Memoryless sched- 
ules with respect to the SUM e objective modeled replication or distribution of copies of resources geared to 
optimize the success probabilities or search times in unstructured p2p networks 0. Our convex program 
formulation extends the solution to a natural situation where each test (resource) is applicable to multiple 
elements (requests). 

Conclusion 

We conducted a comprehensive and unified study of the modeling, algorithmics, and complexity of probe 
scheduling. We reveal inherent gaps between different objectives and between stochastic and deterministic 
schedules and propose efficient scheduling algorithm with analytic performance guarantees. We demonstrate 
the value of this work by simulations on realistic networks. Beyond silent failure detection, we believe the 
optimization problems we address and our scheduling algorithms will find applications in other resource 
allocation domains. 
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A Deferred Proofs 

Proof of Lemma O 

Proof. We replace the original stochastic schedule by a cyclic stochastic schedule cr-N by choosing some 
N and after each repetition of N time steps invoking the original schedule from time 0. We then take a new 
schedule er'-N which uniformly executes one of the N possible shifts of the cyclic schedule cr-N. For each 
element e and for all t, TV_ N (e, t) = E t [e|<r-N], that is, the detection time at any time t on the combined 
shifted schedule is equal to the expected (over time) detection time on the cyclic schedule. 

It remains to show that for any 5 we can choose an N so that for all elements E t [e] on the cyclic schedule 
is within (1 + 5) of E t [e] on the original schedule , that is, E t [e|cr-N] < (1 + 5)E^[e|cr]. 

For each e, there must be N e so that for all h > N e , 

h 

(lA)X)T(e,t)<E t [e](l + e). 
t=i 

We take N to be the maximum of max e N e , max e T(e, l)/e, and max e E t [e]/(|£'|e). 

For each element e, we need to bound the potential increase in expectation over (1/JV) YltLi T{e, t) that 
is due to the "wrap around". Formally, bound Y2t=l T<r-N(e, t) in terms of Ylt=i Ter(e, t). The effect of the 
wrap around is at most that of replacing T(e,t) by T(e, t) + Pr[T(e, t) > N-t]T(e,l). Pr[T(e, t) > 

N-t](N-t) < E t [e](l + e) and therefore Y^t=i Pr[T(e, t) > N-t] < (1 + e)E t [e]. Hence, E^i T ( e >*) 
under the wrap around is at most Y2t=l t) in the original sequence plus T(e, l)Et[e] < eJVEt[e]. Thus 
(1/JV) Y%=i T (e, t) under the wrap around is at most that of the original sequence plus eE t [e\. □ 

Proof of Lemma H3 

Proof. Consider a stochastic schedule <r. E t [e] is well defined for all e and so does the E e E t [<r] = X The 
stochastic schedule is a distribution over deterministic schedules, but a technicality we need to circumvent is 
that the E e E t may not be defined on some of these schedules even when defined for a. For N > 1, we can 
obtain a cyclic stochastic schedule by executing our schedule for the first N time steps, we then discard the 
sequence if there is an element e that is not probed even once, and repeat from the beginning. When we get 
a successful length N sequence, we cycle through it to obtain an infinite schedule. We show that for each 
5 > 0, we can always choose N so that the E e E t of the cyclic schedule is at most (1 + 5)X. This cyclic 
stochastic schedule is a distribution over cyclic deterministic schedules. Since on every cyclic schedule the 
E e E t is well defined, the E e E t of these schedules and also their expected E e E t , which is equal to the E e E t 
of the cyclic stochastic schedule, is well defined. Therefore, there must be a deterministic cyclic schedule 
with E e E t that is at most that of the cyclic stochastic schedule which in turn is at most (1 + 5)X, which 
establishes ([6]). 

It remains to outline how to choose N. For each e, there must be iV e so that for all h > N e (1/h) Ylt=i ^X e > 
E t [e] + e. We take N to be the maximum of max e N e , max e T(e, l)/e, and max e E t [e]/(ne). 

We first observe that the probability that a sequence is discarded is at most e. So by discarding these 
sequences we can not increase the expectation by more than 1/(1 — e) (1 + e). Secondly, we need to 
bound the potential increase in expectation that is due to the "wrap around". This is handled as in the proof 
of Lemma l2~2l 

We now establish the inequalities. Given a deterministic schedule and e, we construct a cyclic determin- 
istic schedule on which the E e M t and M t E e are within (1 + e) of the original deterministic schedule. We 
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take N e S> E e M t /e and such that the E e M t and M t E e on the first h time steps for all h > N e are at most 
(1 + e/2) times that of the original sequence. We now look at the "state", which is the last time since each 
edge was tested and time for next probe. There is a bound L such that the state at most time steps has all 
times at most L. We can now find a subsequence of the schedule that starts and finishes at the same state 
and that the objectives over it are at most (1 + e) times those of the original schedule. We turn it into a cyclic 
schedule. 

We now take this cyclic schedule and construct a stochastic schedule a' by selecting a start point uni- 
formly at random. For each element e, we have 

Ml[eM < "hm±i . 

By combining, 

opt-E e M t < Y,PeMt[e\cr'] < £ Ut ^ + l = (EMt + l)/2 . 

e e ~ 

By taking the minimum E e M t over all deterministic schedules we conclude the claim. The argument for 
M e Mi is similar. □ 



Proof of Lemma 1231 

Proof. We use a reduction to exact cover by sets of size 3 (X3C). We obtain a scheduling instance using 
the same set of elements and subsets (tests) as the X3C instance. We use a uniform p over elements with 
p e = 1/ (3k) for SUM e objectives and p e = 1 for MAX e objectives. 

We first consider deterministic schedules. From an exact cover, we define a deterministic schedule 
by cycling through the same permutation of the cover. The deterministic schedule has M t [e] = k and 
E t [e] = (k + l)/2 for all elements e. The maximum max e T(e, t) at any time t is k and the average is 
(k + 1) /2. Therefore, the schedule has M e M t , E t M e , and E e M t equal to k and M e E 4 , E e E(, and M t E e equal 
to (k + l)/2. 

Consider an arbitrary deterministic schedule and time t. We must have max e T(e, t) > k, since at most 
3i elements can be covered in i probes, so to cover all 3k elements we need at least k probes. We have 
equality if and only if the sequence of k probes following t constitutes a cover. A cover of size k must be an 
exact cover. Therefore E t M e = k implies exact cover of size k. 

Similarly, we claim that on any schedule, (1/k) J2 e T(e, t) > (k + l)/2. This is because ^ e T(e, t) = 
m e , where m e is the smallest d such that e € o~t+d- Since there can be at most 3 elements of each value 
of m e > 1, we have that J2 e t) > 3 J2d=i d = 3k(k + l)/2 and our claim follows. Moreover, equality 
holds only if the sequence of k probes from t on is an exact cover. Therefore M t E e = (k + l)/2 implies 
exact cover of size k. 

Consider an arbitrary deterministic schedule and let q e be the average probing frequency of element e 
(we assume convergence). We have M t [e\ > l/q e and E t [e] > (1 + l/q e )/2. Moreover, equality can hold 
only when l/q e is intergral and probes are evenly spaced every l/q e probes except for vanishingly small 
fraction of times. For the X3C instance we have ^ e q e = 3, and from convexity, ^ e l/q e or max e l/q e 
are minimized only when all q e are equal to 1/k. This means that the M e M t and E e M t can be equal to k 
or the M e Et and E e Et are equal to (k + l)/2 equal to (k + l)/2 only if each element is probed every k 
probes (except vanishingly small) number of times. This means that most sequences of k consecutive probes 
constitute an exact cover. 
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We now consider stochastic schedules. From an exact cover, we define a stochastic schedule by a 
uniform distribution (1/fc) on each of the k shifts of the same permutation of the cover. On this schedule, 
all our objectives have value (k + l)/2. It remains to show that for each of the objectives, a schedule with 
time (k + l)/2 implies an exact cover. 

Observe that with our choice of weighting, on any schedule opt-MAX e > opt-SUM e . Therefore if the 
stochastic optimum of either the SUM e or MAX e objectives is (k + l)/2, then opt-SUM e is also (k + l)/2 
which implies, from that opt£)-E e E t = (k + l)/2, which implies exact cover. □ 



Proof of Lemma l4l2l 

Proof. We choose n, £ > 1, and m > 2£ such that n = (™) , and construct an instance with n elements 
and m tests such that each element is included in exactly I test and every subset of £ tests has exactly one 
common element. We use a unifrom p. 

The instance is symmetric and therefore in the solution of the convex program (|9]> or (10 1 all the m tests 



have equal rates q = 1/m. The memoryless schedule with this q optimizes both SUM e and MAX e for 
p. For any element and any time, the expected detection time by a memoryless schedule with q is m/£. 
But for any particular deterministic schedule and a particular time there is an element that requires m — £ 
probes (for any sequence of m — I tests there must be at least i tests not included and we take the element in 
the intersection of these tests. This means that at any time, the worst-case element detection time is factor 
^rf > £/2 = 0(lnn/ mm) larger than the memoryless optimum. 

When fixing the number of tests m, this is maximized (Sperner's Theorem) with £ = m/2 and the 
MAX e ratio is 0(m). When fixing the number of elements n, the maximum ratio is arg maxf n = Q ) and 
we obtain £ = 0(lnn). 



We use the same construction as in Lemma 4.2 and take a uniform p over elements. Lastly, to show 



E e Mi of Q(ln£) consider a sequence of m probes. The expectation over elements of the number of probes 
that test the element, is at most £. So at least half the elements are probed at most £ times. There are at 
least m/2 distinct tests. The expected over e maximum difference between probes to an element e over a 
sequence of m is Q(ln£)m/£. This is because every combination of £ distinct probes corresponds to an 
element, and thus, for the "average" element, the probes can be viewed as randomly placed, making the 
expectation of the maximum interval a logarithmic factor larger than the expectation. 

We now show how the instances can be realized on a network. We use n pairs of links. Each pair includes 
a link which corresponds to an element in our instance and a "dummy" link. The pairs are connected on a 
path of size n. Each test is an end to end path which traverses one link from each pair. Each (real) link is 
covered by exactly £ paths and every subset of £ paths has one common (real) link. The network is a path of 
length n of pairs of parallel links, a real and a dummy link. Real links e have p e = 1 and the dummy link 
have p e = +oo. (If we want to work with respect to some SUM e optimum we take p e =p (for some p < 1) 
for real links and p e = for dummy links). Each path traverses one link from each pair and includes £ real 
links. □ 

The Lemma is tight in the sense that it is always possible to get a schedule with D2M equal to m by 
cycling over a permutation of the tests. 
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